## [1] "aa_high_acf_loading_shift"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 0.0485 102. 0.961 0.961 ns
## 2 diff1_acf1 150 50 -2.56 76.7 0.0124 0.0201 *
## 3 diff1_acf10 150 50 0.516 83.9 0.607 0.689 ns
## 4 diff2_acf1 150 50 -0.476 76.9 0.636 0.689 ns
## 5 diff2_acf10 150 50 -0.907 77.3 0.367 0.477 ns
## 6 e_acf1 150 50 -2.45 82.4 0.0164 0.0237 *
## 7 e_acf10 150 50 -3.23 66.8 0.00191 0.00429 **
## 8 entropy 150 50 3.21 71.9 0.00198 0.00429 **
## 9 linearity 150 50 -3.47 101. 0.00077 0.00250 **
## 10 spike 150 50 3.98 102. 0.000132 0.000572 ***
## 11 trend 150 50 -4.26 82.4 0.0000542 0.000352 ***
## 12 x_acf1 150 50 -5.08 91.7 0.00000199 0.0000259 ****
## 13 x_acf10 150 50 -3.07 71.3 0.00299 0.00555 **
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_high_acf_loading_shift | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.790 | 0.970 | 0.130 | 0.930 | 0.940 | 0.940 | 0.200 | 0.980 | 0.850 | 0.230 |
| 2 | 0.700 | 0.800 | 0.040 | 0.760 | 0.760 | 0.880 | 0.100 | 0.970 | 0.750 | 0.120 |
| 3 | 0.600 | 0.680 | 0.070 | 0.570 | 0.660 | 0.860 | 0.090 | 1.000 | 0.690 | 0.060 |
| 4 | 0.660 | 0.810 | 0.060 | 0.770 | 0.850 | 0.860 | 0.100 | 0.980 | 0.860 | 0.080 |
| rmse | ||||||||||
| 0 | 0.045 | 0.034 | 0.099 | 0.036 | 0.033 | 0.039 | 0.070 | 0.070 | 0.047 | 0.090 |
| 1 | 0.059 | 0.041 | 0.095 | 0.046 | 0.044 | 0.043 | 0.073 | 0.052 | 0.056 | 0.109 |
| 2 | 0.069 | 0.060 | 0.117 | 0.065 | 0.063 | 0.055 | 0.092 | 0.042 | 0.065 | 0.127 |
| 3 | 0.077 | 0.071 | 0.125 | 0.077 | 0.073 | 0.057 | 0.089 | 0.035 | 0.073 | 0.136 |
| 4 | 0.074 | 0.060 | 0.120 | 0.061 | 0.052 | 0.054 | 0.088 | 0.054 | 0.054 | 0.123 |
| bias | ||||||||||
| 0 | −0.030 | −0.011 | −0.043 | −0.017 | −0.009 | −0.004 | −0.011 | 0.061 | −0.027 | −0.082 |
| 1 | −0.049 | −0.029 | −0.048 | −0.036 | −0.032 | −0.021 | −0.017 | 0.044 | −0.042 | −0.103 |
| 2 | −0.058 | −0.050 | −0.074 | −0.055 | −0.052 | −0.039 | −0.051 | 0.026 | −0.053 | −0.121 |
| 3 | −0.071 | −0.064 | −0.086 | −0.071 | −0.066 | −0.044 | −0.055 | 0.018 | −0.063 | −0.132 |
| 4 | −0.068 | −0.053 | −0.078 | −0.055 | −0.044 | −0.041 | −0.060 | 0.046 | −0.043 | −0.119 |
| Notes: | ||||||||||
## [1] "aa_high_acf"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 -0.544 75.7 0.588 0.961 ns
## 2 diff1_acf1 150 50 -0.269 78.4 0.789 0.961 ns
## 3 diff1_acf10 150 50 -0.164 71.1 0.87 0.961 ns
## 4 diff2_acf1 150 50 -0.606 74.1 0.546 0.961 ns
## 5 diff2_acf10 150 50 0.0230 70.7 0.982 0.982 ns
## 6 e_acf1 150 50 0.284 75.4 0.777 0.961 ns
## 7 e_acf10 150 50 0.180 96.0 0.858 0.961 ns
## 8 entropy 150 50 1.71 68.4 0.0922 0.599 ns
## 9 linearity 150 50 -0.560 70.4 0.577 0.961 ns
## 10 spike 150 50 0.143 78.7 0.887 0.961 ns
## 11 trend 150 50 -0.902 73.3 0.37 0.961 ns
## 12 x_acf1 150 50 -1.07 74.1 0.290 0.961 ns
## 13 x_acf10 150 50 -1.80 67.5 0.0765 0.599 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_high_acf | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.980 | 0.990 | 0.070 | 0.860 | 0.430 | 0.860 | 0.160 | 0.000 | 0.930 | 0.860 |
| 2 | 0.980 | 1.000 | 0.050 | 0.980 | 0.700 | 0.980 | 0.140 | 0.000 | 0.840 | 0.950 |
| 3 | 0.960 | 0.980 | 0.080 | 0.830 | 0.620 | 0.890 | 0.150 | 0.000 | 0.880 | 0.890 |
| 4 | 0.980 | 0.980 | 0.090 | 0.780 | 0.520 | 0.850 | 0.130 | 0.000 | 0.920 | 0.940 |
| rmse | ||||||||||
| 0 | 0.031 | 0.030 | 0.106 | 0.046 | 0.078 | 0.044 | 0.066 | 0.197 | 0.034 | 0.041 |
| 1 | 0.031 | 0.030 | 0.123 | 0.046 | 0.079 | 0.051 | 0.065 | 0.219 | 0.038 | 0.044 |
| 2 | 0.028 | 0.025 | 0.122 | 0.035 | 0.060 | 0.031 | 0.064 | 0.226 | 0.046 | 0.037 |
| 3 | 0.035 | 0.033 | 0.137 | 0.048 | 0.073 | 0.043 | 0.066 | 0.262 | 0.048 | 0.045 |
| 4 | 0.034 | 0.036 | 0.130 | 0.060 | 0.078 | 0.056 | 0.075 | 0.293 | 0.040 | 0.041 |
| bias | ||||||||||
| 0 | −0.004 | 0.007 | −0.032 | 0.035 | 0.072 | 0.028 | −0.001 | 0.195 | −0.011 | 0.004 |
| 1 | 0.002 | 0.006 | −0.042 | 0.036 | 0.073 | 0.034 | −0.001 | 0.217 | −0.018 | 0.005 |
| 2 | −0.009 | −0.010 | −0.055 | 0.025 | 0.055 | 0.008 | −0.015 | 0.225 | −0.036 | −0.014 |
| 3 | −0.003 | −0.001 | −0.035 | 0.036 | 0.064 | 0.020 | 0.011 | 0.260 | −0.032 | −0.004 |
| 4 | 0.002 | 0.018 | −0.003 | 0.052 | 0.073 | 0.043 | 0.034 | 0.291 | −0.020 | 0.006 |
| Notes: | ||||||||||
## [1] "aa_low_acf_int_shift"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 -1.57 128. 0.118 0.612 ns
## 2 diff1_acf1 150 50 -0.657 83.0 0.513 0.667 ns
## 3 diff1_acf10 150 50 -0.804 95.7 0.423 0.612 ns
## 4 diff2_acf1 150 50 0.0291 89.9 0.977 0.977 ns
## 5 diff2_acf10 150 50 -0.803 101. 0.424 0.612 ns
## 6 e_acf1 150 50 -0.969 87.1 0.335 0.612 ns
## 7 e_acf10 150 50 -1.30 77.8 0.197 0.612 ns
## 8 entropy 150 50 1.30 81.7 0.197 0.612 ns
## 9 linearity 150 50 0.340 89.1 0.734 0.795 ns
## 10 spike 150 50 1.09 88.5 0.279 0.612 ns
## 11 trend 150 50 0.901 109. 0.369 0.612 ns
## 12 x_acf1 150 50 -0.468 89.5 0.641 0.758 ns
## 13 x_acf10 150 50 -1.05 82.5 0.295 0.612 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_low_acf_int_shift | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.970 | 0.880 | 0.130 | 0.890 | 0.990 | 0.930 | 0.140 | 0.940 | 1.000 | 0.880 |
| 2 | 0.940 | 0.880 | 0.120 | 0.870 | 0.940 | 0.940 | 0.130 | 0.960 | 0.940 | 0.870 |
| 3 | 0.950 | 0.970 | 0.190 | 0.950 | 0.950 | 0.930 | 0.240 | 0.890 | 0.880 | 0.920 |
| 4 | 0.970 | 0.980 | 0.120 | 0.970 | 0.970 | 1.000 | 0.190 | 0.950 | 0.960 | 0.930 |
| rmse | ||||||||||
| 0 | 0.031 | 0.035 | 0.063 | 0.033 | 0.030 | 0.037 | 0.055 | 0.032 | 0.032 | 0.035 |
| 1 | 0.031 | 0.039 | 0.074 | 0.039 | 0.031 | 0.042 | 0.064 | 0.036 | 0.033 | 0.039 |
| 2 | 0.032 | 0.040 | 0.066 | 0.040 | 0.034 | 0.038 | 0.063 | 0.034 | 0.036 | 0.040 |
| 3 | 0.030 | 0.030 | 0.058 | 0.029 | 0.032 | 0.037 | 0.059 | 0.039 | 0.037 | 0.031 |
| 4 | 0.028 | 0.027 | 0.054 | 0.027 | 0.028 | 0.028 | 0.056 | 0.034 | 0.029 | 0.028 |
| bias | ||||||||||
| 0 | 0.003 | −0.016 | −0.026 | −0.011 | 0.002 | −0.011 | −0.011 | 0.009 | 0.005 | −0.013 |
| 1 | −0.007 | −0.026 | −0.026 | −0.025 | 0.010 | −0.025 | −0.022 | 0.023 | −0.003 | −0.024 |
| 2 | −0.005 | −0.024 | −0.034 | −0.023 | −0.005 | −0.017 | −0.016 | 0.012 | −0.007 | −0.024 |
| 3 | 0.002 | 0.002 | −0.005 | 0.004 | 0.012 | 0.021 | 0.001 | 0.026 | 0.012 | 0.003 |
| 4 | 0.004 | −0.005 | 0.006 | −0.004 | 0.007 | 0.002 | 0.002 | 0.022 | 0.005 | −0.006 |
| Notes: | ||||||||||
## [1] "aa_low_acf_load_int_shifts"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 0.809 110. 0.42 0.42 ns
## 2 diff1_acf1 150 50 -4.23 94.2 0.0000533 0.000115 ***
## 3 diff1_acf10 150 50 2.89 146. 0.00448 0.00529 **
## 4 diff2_acf1 150 50 -3.51 94.4 0.000683 0.000987 ***
## 5 diff2_acf10 150 50 3.58 152. 0.00047 0.000764 ***
## 6 e_acf1 150 50 -5.42 107. 0.000000378 0.00000246 ****
## 7 e_acf10 150 50 -4.89 70.6 0.00000617 0.0000267 ****
## 8 entropy 150 50 4.67 79.3 0.0000121 0.0000393 ****
## 9 linearity 150 50 -1.71 92.1 0.091 0.0986 ns
## 10 spike 150 50 3.13 97.0 0.00228 0.00296 **
## 11 trend 150 50 -4.48 100. 0.0000197 0.0000512 ****
## 12 x_acf1 150 50 -5.83 114. 0.0000000532 0.000000692 ****
## 13 x_acf10 150 50 -4.04 85.8 0.000118 0.000219 ***
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_low_acf_load_int_shifts | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.950 | 0.930 | 0.150 | 0.920 | 0.960 | 0.930 | 0.190 | 0.810 | 0.890 | 0.930 |
| 2 | 0.970 | 0.970 | 0.250 | 0.960 | 0.970 | 0.980 | 0.180 | 0.920 | 0.960 | 0.960 |
| 3 | 0.940 | 0.850 | 0.220 | 0.820 | 0.760 | 0.890 | 0.180 | 0.420 | 0.900 | 0.840 |
| 4 | 0.950 | 0.920 | 0.190 | 0.920 | 0.740 | 0.900 | 0.160 | 0.250 | 0.930 | 0.920 |
| rmse | ||||||||||
| 0 | 0.034 | 0.033 | 0.066 | 0.033 | 0.033 | 0.042 | 0.060 | 0.044 | 0.040 | 0.033 |
| 1 | 0.033 | 0.036 | 0.057 | 0.036 | 0.036 | 0.037 | 0.063 | 0.057 | 0.042 | 0.035 |
| 2 | 0.027 | 0.033 | 0.056 | 0.032 | 0.032 | 0.033 | 0.061 | 0.043 | 0.035 | 0.034 |
| 3 | 0.032 | 0.039 | 0.052 | 0.038 | 0.053 | 0.044 | 0.062 | 0.084 | 0.043 | 0.040 |
| 4 | 0.031 | 0.033 | 0.067 | 0.032 | 0.054 | 0.051 | 0.062 | 0.098 | 0.038 | 0.033 |
| bias | ||||||||||
| 0 | 0.000 | 0.008 | 0.004 | 0.009 | 0.003 | −0.023 | −0.010 | 0.030 | 0.013 | 0.006 |
| 1 | 0.007 | 0.020 | 0.015 | 0.018 | 0.016 | −0.007 | 0.002 | 0.048 | 0.019 | 0.015 |
| 2 | 0.003 | 0.020 | 0.010 | 0.020 | 0.017 | 0.010 | 0.005 | 0.034 | 0.017 | 0.019 |
| 3 | 0.005 | 0.023 | 0.006 | 0.023 | 0.043 | 0.028 | −0.010 | 0.079 | 0.024 | 0.025 |
| 4 | −0.006 | 0.013 | 0.002 | 0.012 | 0.044 | −0.034 | −0.022 | 0.094 | 0.016 | 0.008 |
| Notes: | ||||||||||
## [1] "aa_low_acf"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 -1.83 83.1 0.0713 0.927 ns
## 2 diff1_acf1 150 50 -0.463 91.5 0.644 0.989 ns
## 3 diff1_acf10 150 50 0.209 85.5 0.835 0.989 ns
## 4 diff2_acf1 150 50 -0.168 90.2 0.867 0.989 ns
## 5 diff2_acf10 150 50 0.358 97.0 0.721 0.989 ns
## 6 e_acf1 150 50 0.0134 87.8 0.989 0.989 ns
## 7 e_acf10 150 50 -0.686 81.8 0.495 0.989 ns
## 8 entropy 150 50 0.541 74.0 0.59 0.989 ns
## 9 linearity 150 50 0.981 81.5 0.329 0.989 ns
## 10 spike 150 50 0.640 81.0 0.524 0.989 ns
## 11 trend 150 50 0.299 123. 0.765 0.989 ns
## 12 x_acf1 150 50 0.0152 96.3 0.988 0.989 ns
## 13 x_acf10 150 50 -0.261 93.5 0.794 0.989 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_low_acf | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.970 | 0.930 | 0.170 | 0.940 | 0.990 | 0.910 | 0.160 | 0.950 | 1.000 | 0.920 |
| 2 | 0.940 | 0.900 | 0.120 | 0.910 | 0.940 | 0.930 | 0.140 | 0.950 | 0.950 | 0.850 |
| 3 | 0.950 | 0.970 | 0.150 | 0.960 | 0.950 | 0.910 | 0.190 | 0.890 | 0.880 | 0.960 |
| 4 | 0.970 | 0.980 | 0.160 | 0.980 | 0.970 | 1.000 | 0.170 | 0.950 | 0.960 | 0.960 |
| rmse | ||||||||||
| 0 | 0.031 | 0.032 | 0.063 | 0.032 | 0.031 | 0.038 | 0.062 | 0.032 | 0.031 | 0.036 |
| 1 | 0.031 | 0.037 | 0.056 | 0.036 | 0.031 | 0.043 | 0.062 | 0.036 | 0.034 | 0.039 |
| 2 | 0.032 | 0.039 | 0.061 | 0.037 | 0.034 | 0.038 | 0.064 | 0.034 | 0.036 | 0.042 |
| 3 | 0.030 | 0.030 | 0.061 | 0.030 | 0.032 | 0.037 | 0.063 | 0.039 | 0.037 | 0.029 |
| 4 | 0.028 | 0.027 | 0.052 | 0.027 | 0.028 | 0.027 | 0.063 | 0.034 | 0.030 | 0.027 |
| bias | ||||||||||
| 0 | 0.003 | −0.011 | −0.017 | −0.007 | 0.002 | −0.012 | −0.008 | 0.009 | 0.005 | −0.013 |
| 1 | −0.007 | −0.022 | −0.017 | −0.020 | 0.010 | −0.026 | −0.015 | 0.022 | −0.002 | −0.024 |
| 2 | −0.005 | −0.022 | −0.012 | −0.019 | −0.005 | −0.018 | −0.003 | 0.011 | −0.007 | −0.024 |
| 3 | 0.002 | 0.008 | −0.006 | 0.009 | 0.012 | 0.022 | −0.003 | 0.026 | 0.012 | 0.004 |
| 4 | 0.005 | −0.000 | −0.010 | 0.003 | 0.006 | 0.001 | 0.009 | 0.022 | 0.005 | −0.004 |
| Notes: | ||||||||||
## [1] "aa_noisy_factors_load_shift_lowacf"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 -2.64 104. 0.00947 0.0246 *
## 2 diff1_acf1 150 50 1.54 81.9 0.128 0.185 ns
## 3 diff1_acf10 150 50 -3.90 79.1 0.000198 0.00129 **
## 4 diff2_acf1 150 50 1.55 101. 0.123 0.185 ns
## 5 diff2_acf10 150 50 -4.74 83.8 0.00000861 0.000112 ***
## 6 e_acf1 150 50 0.868 77.8 0.388 0.459 ns
## 7 e_acf10 150 50 -3.43 82.6 0.000937 0.00305 **
## 8 entropy 150 50 3.78 64.8 0.000339 0.00147 **
## 9 linearity 150 50 0.439 92.7 0.662 0.662 ns
## 10 spike 150 50 1.25 84.7 0.216 0.281 ns
## 11 trend 150 50 -1.66 65.9 0.102 0.185 ns
## 12 x_acf1 150 50 -0.524 70.5 0.602 0.652 ns
## 13 x_acf10 150 50 -1.69 51.9 0.0975 0.185 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_noisy_factors_load_shift_lowacf | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.910 | 0.930 | 0.150 | 0.920 | 0.940 | 0.910 | 0.190 | 0.960 | 0.910 | 0.940 |
| 2 | 0.940 | 0.940 | 0.130 | 0.920 | 0.970 | 0.960 | 0.180 | 0.990 | 0.980 | 0.920 |
| 3 | 0.940 | 0.930 | 0.190 | 0.940 | 0.940 | 0.950 | 0.220 | 0.950 | 0.910 | 0.950 |
| 4 | 0.970 | 0.980 | 0.190 | 0.970 | 0.990 | 0.950 | 0.200 | 0.950 | 0.960 | 0.960 |
| rmse | ||||||||||
| 0 | 0.035 | 0.037 | 0.057 | 0.037 | 0.035 | 0.037 | 0.064 | 0.033 | 0.037 | 0.033 |
| 1 | 0.035 | 0.037 | 0.057 | 0.036 | 0.034 | 0.045 | 0.060 | 0.031 | 0.038 | 0.034 |
| 2 | 0.033 | 0.039 | 0.067 | 0.039 | 0.033 | 0.036 | 0.064 | 0.029 | 0.034 | 0.034 |
| 3 | 0.032 | 0.035 | 0.057 | 0.036 | 0.032 | 0.039 | 0.050 | 0.033 | 0.040 | 0.034 |
| 4 | 0.030 | 0.030 | 0.061 | 0.030 | 0.029 | 0.043 | 0.053 | 0.034 | 0.035 | 0.031 |
| bias | ||||||||||
| 0 | −0.014 | −0.022 | −0.021 | −0.021 | −0.012 | 0.008 | −0.003 | 0.009 | −0.011 | −0.007 |
| 1 | −0.016 | −0.020 | −0.023 | −0.019 | −0.011 | −0.025 | −0.014 | −0.002 | −0.008 | −0.010 |
| 2 | −0.009 | −0.026 | −0.026 | −0.027 | −0.017 | 0.001 | −0.000 | 0.003 | −0.006 | −0.013 |
| 3 | −0.009 | −0.016 | −0.016 | −0.018 | −0.001 | −0.011 | −0.010 | 0.011 | 0.001 | −0.007 |
| 4 | −0.008 | −0.003 | −0.003 | −0.006 | 0.001 | 0.029 | 0.015 | 0.018 | 0.004 | 0.008 |
| Notes: | ||||||||||
## [1] "aa_noisy_factors_load_shift"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 -0.0244 93.7 0.981 0.981 ns
## 2 diff1_acf1 150 50 -5.14 81.5 0.00000182 0.0000195 ****
## 3 diff1_acf10 150 50 4.74 105. 0.00000665 0.0000288 ****
## 4 diff2_acf1 150 50 -3.13 89.5 0.00233 0.00303 **
## 5 diff2_acf10 150 50 2.97 105. 0.00371 0.00402 **
## 6 e_acf1 150 50 -4.46 96.7 0.0000225 0.0000581 ****
## 7 e_acf10 150 50 -3.78 74.3 0.000314 0.000583 ***
## 8 entropy 150 50 3.03 69.5 0.00341 0.00402 **
## 9 linearity 150 50 -3.70 89.6 0.000371 0.000603 ***
## 10 spike 150 50 4.91 118. 0.000003 0.0000195 ****
## 11 trend 150 50 -4.44 84.4 0.0000268 0.0000581 ****
## 12 x_acf1 150 50 -4.59 84.2 0.0000156 0.0000507 ****
## 13 x_acf10 150 50 -3.17 66.8 0.00231 0.00303 **
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_noisy_factors_load_shift | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.900 | 0.070 | 0.060 | 0.090 | 0.000 | 0.570 | 0.070 | 0.000 | 0.880 | 0.970 |
| 2 | 0.950 | 0.100 | 0.100 | 0.080 | 0.010 | 0.730 | 0.230 | 0.000 | 0.900 | 0.900 |
| 3 | 0.980 | 0.050 | 0.050 | 0.040 | 0.000 | 0.660 | 0.160 | 0.000 | 0.910 | 0.940 |
| 4 | 0.960 | 0.110 | 0.090 | 0.070 | 0.010 | 0.610 | 0.120 | 0.000 | 0.860 | 0.890 |
| rmse | ||||||||||
| 0 | 0.042 | 0.164 | 0.138 | 0.157 | 0.177 | 0.086 | 0.090 | 0.361 | 0.062 | 0.043 |
| 1 | 0.047 | 0.158 | 0.137 | 0.153 | 0.176 | 0.086 | 0.090 | 0.394 | 0.057 | 0.042 |
| 2 | 0.038 | 0.157 | 0.137 | 0.156 | 0.185 | 0.079 | 0.085 | 0.439 | 0.054 | 0.047 |
| 3 | 0.041 | 0.171 | 0.139 | 0.170 | 0.194 | 0.087 | 0.091 | 0.513 | 0.058 | 0.051 |
| 4 | 0.047 | 0.169 | 0.153 | 0.172 | 0.197 | 0.098 | 0.105 | 0.548 | 0.057 | 0.057 |
| bias | ||||||||||
| 0 | 0.028 | 0.162 | 0.095 | 0.154 | 0.174 | 0.079 | 0.046 | 0.359 | 0.050 | 0.018 |
| 1 | 0.033 | 0.154 | 0.085 | 0.149 | 0.172 | 0.078 | 0.049 | 0.392 | 0.042 | 0.016 |
| 2 | 0.019 | 0.153 | 0.087 | 0.153 | 0.181 | 0.069 | 0.046 | 0.438 | 0.037 | 0.025 |
| 3 | 0.024 | 0.168 | 0.095 | 0.168 | 0.191 | 0.079 | 0.039 | 0.512 | 0.043 | 0.034 |
| 4 | 0.028 | 0.166 | 0.101 | 0.169 | 0.193 | 0.088 | 0.042 | 0.547 | 0.038 | 0.037 |
| Notes: | ||||||||||
## [1] "aa_noisy_factors_lowacf"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 1.69 76.2 0.0946 0.679 ns
## 2 diff1_acf1 150 50 0.336 78.7 0.738 0.881 ns
## 3 diff1_acf10 150 50 -0.237 85.2 0.813 0.881 ns
## 4 diff2_acf1 150 50 1.50 74.9 0.137 0.679 ns
## 5 diff2_acf10 150 50 -1.21 86.0 0.231 0.679 ns
## 6 e_acf1 150 50 -0.725 87.7 0.47 0.679 ns
## 7 e_acf10 150 50 -0.325 93.5 0.746 0.881 ns
## 8 entropy 150 50 0.829 60.1 0.41 0.679 ns
## 9 linearity 150 50 -0.0488 74.9 0.961 0.961 ns
## 10 spike 150 50 0.799 86.7 0.426 0.679 ns
## 11 trend 150 50 -0.806 67.5 0.423 0.679 ns
## 12 x_acf1 150 50 -0.755 72.1 0.453 0.679 ns
## 13 x_acf10 150 50 -1.11 51.8 0.273 0.679 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_noisy_factors_lowacf | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.980 | 0.980 | 0.190 | 0.990 | 1.000 | 0.970 | 0.160 | 0.900 | 0.980 | 0.980 |
| 2 | 0.950 | 0.980 | 0.170 | 0.990 | 0.980 | 0.980 | 0.150 | 0.880 | 0.990 | 0.970 |
| 3 | 0.950 | 0.970 | 0.130 | 0.960 | 0.880 | 0.960 | 0.250 | 0.620 | 0.950 | 0.960 |
| 4 | 0.960 | 0.960 | 0.160 | 0.960 | 0.850 | 0.950 | 0.120 | 0.740 | 0.920 | 0.960 |
| rmse | ||||||||||
| 0 | 0.031 | 0.032 | 0.054 | 0.031 | 0.038 | 0.039 | 0.063 | 0.052 | 0.033 | 0.032 |
| 1 | 0.027 | 0.027 | 0.059 | 0.026 | 0.027 | 0.032 | 0.058 | 0.041 | 0.030 | 0.027 |
| 2 | 0.031 | 0.030 | 0.063 | 0.030 | 0.031 | 0.032 | 0.061 | 0.043 | 0.030 | 0.032 |
| 3 | 0.035 | 0.032 | 0.058 | 0.032 | 0.043 | 0.034 | 0.060 | 0.059 | 0.038 | 0.032 |
| 4 | 0.033 | 0.032 | 0.069 | 0.032 | 0.042 | 0.037 | 0.069 | 0.058 | 0.038 | 0.033 |
| bias | ||||||||||
| 0 | −0.001 | 0.011 | 0.002 | 0.011 | 0.024 | 0.016 | 0.009 | 0.043 | 0.012 | 0.007 |
| 1 | 0.004 | 0.002 | 0.001 | 0.004 | 0.008 | 0.011 | 0.011 | 0.032 | 0.004 | −0.001 |
| 2 | 0.001 | −0.002 | −0.011 | −0.001 | 0.008 | −0.002 | −0.012 | 0.032 | −0.001 | −0.006 |
| 3 | −0.000 | 0.004 | −0.012 | 0.005 | 0.030 | 0.000 | 0.003 | 0.051 | 0.013 | 0.001 |
| 4 | 0.004 | 0.006 | 0.012 | 0.009 | 0.028 | 0.009 | 0.011 | 0.049 | 0.011 | 0.005 |
| Notes: | ||||||||||
## [1] "aa_noisy_factors"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 0.786 80.4 0.434 0.627 ns
## 2 diff1_acf1 150 50 -0.549 81.4 0.584 0.690 ns
## 3 diff1_acf10 150 50 1.47 84.6 0.145 0.269 ns
## 4 diff2_acf1 150 50 0.0853 87.2 0.932 0.944 ns
## 5 diff2_acf10 150 50 0.857 80.1 0.394 0.627 ns
## 6 e_acf1 150 50 0.0708 80.3 0.944 0.944 ns
## 7 e_acf10 150 50 -0.643 69.0 0.522 0.679 ns
## 8 entropy 150 50 2.11 72.2 0.0382 0.0938 ns
## 9 linearity 150 50 -2.21 78.3 0.03 0.0938 ns
## 10 spike 150 50 2.11 90.8 0.0372 0.0938 ns
## 11 trend 150 50 -2.40 83.7 0.0184 0.0938 ns
## 12 x_acf1 150 50 -2.10 82.5 0.0385 0.0938 ns
## 13 x_acf10 150 50 -2.06 75.1 0.0433 0.0938 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| aa_noisy_factors | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.900 | 0.940 | 0.060 | 0.840 | 0.790 | 0.840 | 0.130 | 0.030 | 0.950 | 0.030 |
| 2 | 0.900 | 0.960 | 0.090 | 0.910 | 0.940 | 0.850 | 0.150 | 0.050 | 0.940 | 0.000 |
| 3 | 0.910 | 0.960 | 0.100 | 0.900 | 0.770 | 0.830 | 0.180 | 0.000 | 0.960 | 0.030 |
| 4 | 0.860 | 0.870 | 0.090 | 0.800 | 0.430 | 0.670 | 0.100 | 0.000 | 0.910 | 0.110 |
| rmse | ||||||||||
| 0 | 0.034 | 0.031 | 0.099 | 0.038 | 0.038 | 0.046 | 0.073 | 0.158 | 0.032 | 0.202 |
| 1 | 0.042 | 0.039 | 0.101 | 0.050 | 0.062 | 0.054 | 0.078 | 0.191 | 0.037 | 0.178 |
| 2 | 0.042 | 0.035 | 0.109 | 0.044 | 0.043 | 0.054 | 0.079 | 0.192 | 0.041 | 0.204 |
| 3 | 0.038 | 0.034 | 0.102 | 0.048 | 0.072 | 0.058 | 0.069 | 0.248 | 0.035 | 0.164 |
| 4 | 0.046 | 0.049 | 0.106 | 0.060 | 0.097 | 0.075 | 0.074 | 0.280 | 0.043 | 0.143 |
| bias | ||||||||||
| 0 | 0.010 | 0.012 | 0.001 | 0.026 | 0.026 | 0.031 | 0.008 | 0.155 | −0.004 | −0.199 |
| 1 | 0.021 | 0.020 | −0.003 | 0.038 | 0.053 | 0.040 | 0.012 | 0.188 | 0.007 | −0.173 |
| 2 | 0.016 | 0.010 | 0.006 | 0.028 | 0.025 | 0.039 | 0.003 | 0.189 | −0.014 | −0.199 |
| 3 | 0.021 | 0.019 | 0.011 | 0.039 | 0.067 | 0.048 | 0.019 | 0.246 | 0.011 | −0.159 |
| 4 | 0.022 | 0.035 | 0.018 | 0.050 | 0.091 | 0.064 | 0.020 | 0.278 | 0.020 | −0.136 |
| Notes: | ||||||||||
## [1] "ab_decay_het_loading_shift"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 1.39 96.4 1.68e- 1 1.99e- 1 ns
## 2 diff1_acf1 150 50 -7.51 72.2 1.22e-10 5.29e-10 ****
## 3 diff1_acf10 150 50 1.96 124. 5.25e- 2 9.75e- 2 ns
## 4 diff2_acf1 150 50 -3.73 70.4 3.85e- 4 1.00e- 3 **
## 5 diff2_acf10 150 50 1.83 87.1 7.01e- 2 1.06e- 1 ns
## 6 e_acf1 150 50 -8.78 91.1 9.01e-14 1.17e-12 ****
## 7 e_acf10 150 50 -7.90 68.3 3.31e-11 2.15e-10 ****
## 8 entropy 150 50 1.60 113. 1.13e- 1 1.47e- 1 ns
## 9 linearity 150 50 -0.370 109. 7.12e- 1 7.71e- 1 ns
## 10 spike 150 50 2.16 97.0 3.32e- 2 7.19e- 2 ns
## 11 trend 150 50 -1.81 111. 7.37e- 2 1.06e- 1 ns
## 12 x_acf1 150 50 -4.20 110. 5.48e- 5 1.78e- 4 ***
## 13 x_acf10 150 50 -0.161 159. 8.72e- 1 8.72e- 1 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ab_decay_het_loading_shift | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.970 | 1.000 | 0.170 | 1.000 | 0.970 | 0.980 | 0.190 | 0.940 | 0.970 | 0.960 |
| 2 | 0.950 | 1.000 | 0.180 | 1.000 | 0.990 | 0.990 | 0.160 | 0.870 | 1.000 | 0.950 |
| 3 | 0.910 | 0.970 | 0.150 | 0.970 | 0.970 | 1.000 | 0.150 | 0.970 | 0.980 | 0.980 |
| 4 | 0.900 | 0.990 | 0.150 | 0.980 | 0.990 | 0.970 | 0.100 | 0.970 | 0.990 | 0.990 |
| rmse | ||||||||||
| 0 | 0.035 | 0.031 | 0.065 | 0.033 | 0.048 | 0.038 | 0.068 | 0.070 | 0.038 | 0.050 |
| 1 | 0.037 | 0.031 | 0.079 | 0.029 | 0.034 | 0.036 | 0.069 | 0.055 | 0.036 | 0.038 |
| 2 | 0.038 | 0.031 | 0.074 | 0.030 | 0.033 | 0.038 | 0.075 | 0.057 | 0.034 | 0.040 |
| 3 | 0.043 | 0.039 | 0.076 | 0.034 | 0.030 | 0.034 | 0.069 | 0.044 | 0.037 | 0.032 |
| 4 | 0.040 | 0.036 | 0.080 | 0.033 | 0.030 | 0.038 | 0.081 | 0.044 | 0.035 | 0.033 |
| bias | ||||||||||
| 0 | −0.014 | 0.005 | 0.026 | 0.011 | 0.036 | 0.011 | 0.018 | 0.064 | 0.009 | 0.037 |
| 1 | −0.022 | −0.009 | 0.034 | −0.003 | 0.016 | 0.012 | 0.018 | 0.047 | −0.003 | 0.021 |
| 2 | −0.021 | −0.006 | 0.035 | −0.001 | 0.009 | 0.001 | 0.039 | 0.049 | −0.004 | 0.019 |
| 3 | −0.029 | −0.026 | 0.027 | −0.019 | −0.003 | −0.008 | 0.015 | 0.034 | −0.016 | 0.004 |
| 4 | −0.025 | −0.022 | 0.034 | −0.016 | −0.001 | −0.006 | 0.031 | 0.034 | −0.013 | 0.006 |
| Notes: | ||||||||||
## [1] "ab_decay_het"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 -1.32 91.7 0.189 0.672 ns
## 2 diff1_acf1 150 50 0.0652 84.7 0.948 0.948 ns
## 3 diff1_acf10 150 50 0.560 118. 0.577 0.682 ns
## 4 diff2_acf1 150 50 -0.565 90.4 0.574 0.682 ns
## 5 diff2_acf10 150 50 1.06 111. 0.293 0.672 ns
## 6 e_acf1 150 50 0.0777 81.6 0.938 0.948 ns
## 7 e_acf10 150 50 1.13 82.7 0.261 0.672 ns
## 8 entropy 150 50 -0.734 78.0 0.465 0.672 ns
## 9 linearity 150 50 1.06 81.5 0.292 0.672 ns
## 10 spike 150 50 -0.904 82.1 0.369 0.672 ns
## 11 trend 150 50 0.942 90.2 0.349 0.672 ns
## 12 x_acf1 150 50 0.739 91.4 0.462 0.672 ns
## 13 x_acf10 150 50 0.895 90.0 0.373 0.672 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ab_decay_het | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.950 | 0.940 | 0.180 | 0.980 | 0.880 | 0.850 | 0.170 | 0.840 | 0.920 | 0.880 |
| 2 | 0.970 | 0.990 | 0.090 | 1.000 | 0.600 | 0.940 | 0.150 | 0.390 | 0.970 | 0.700 |
| 3 | 0.930 | 0.900 | 0.020 | 0.890 | 0.330 | 0.960 | 0.160 | 0.110 | 0.950 | 0.590 |
| 4 | 0.960 | 0.960 | 0.120 | 0.940 | 0.560 | 0.980 | 0.090 | 0.200 | 0.960 | 0.660 |
| rmse | ||||||||||
| 0 | 0.033 | 0.036 | 0.074 | 0.032 | 0.043 | 0.050 | 0.072 | 0.051 | 0.037 | 0.041 |
| 1 | 0.033 | 0.039 | 0.066 | 0.032 | 0.041 | 0.049 | 0.068 | 0.057 | 0.039 | 0.039 |
| 2 | 0.030 | 0.028 | 0.086 | 0.030 | 0.072 | 0.038 | 0.083 | 0.095 | 0.033 | 0.059 |
| 3 | 0.034 | 0.038 | 0.101 | 0.041 | 0.091 | 0.036 | 0.094 | 0.124 | 0.037 | 0.073 |
| 4 | 0.033 | 0.036 | 0.103 | 0.040 | 0.080 | 0.036 | 0.099 | 0.120 | 0.035 | 0.069 |
| bias | ||||||||||
| 0 | −0.001 | 0.017 | −0.032 | 0.006 | −0.030 | 0.037 | −0.026 | −0.040 | 0.016 | −0.021 |
| 1 | 0.010 | 0.027 | −0.014 | 0.015 | −0.030 | 0.037 | −0.010 | −0.050 | 0.020 | −0.021 |
| 2 | 0.001 | −0.003 | −0.040 | −0.011 | −0.065 | 0.018 | −0.045 | −0.091 | −0.002 | −0.049 |
| 3 | −0.011 | −0.024 | −0.074 | −0.029 | −0.086 | 0.016 | −0.061 | −0.120 | −0.018 | −0.062 |
| 4 | −0.004 | −0.018 | −0.068 | −0.024 | −0.073 | 0.008 | −0.065 | −0.116 | −0.009 | −0.058 |
| Notes: | ||||||||||
## [1] "ab_decay_impact_het_loading_shift"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 1.24 98.5 2.18e- 1 0.258 ns
## 2 diff1_acf1 150 50 -6.05 82.6 4.11e- 8 0.000000178 ****
## 3 diff1_acf10 150 50 2.67 147. 8.33e- 3 0.0180 *
## 4 diff2_acf1 150 50 -1.79 79.8 7.68e- 2 0.125 ns
## 5 diff2_acf10 150 50 1.48 104. 1.42e- 1 0.185 ns
## 6 e_acf1 150 50 -7.20 92.0 1.60e-10 0.00000000208 ****
## 7 e_acf10 150 50 -7.25 68.8 4.70e-10 0.00000000306 ****
## 8 entropy 150 50 1.70 127. 9.12e- 2 0.132 ns
## 9 linearity 150 50 -0.385 109. 7.01e- 1 0.759 ns
## 10 spike 150 50 3.05 127. 2.81e- 3 0.00731 **
## 11 trend 150 50 -2.21 119. 2.88e- 2 0.0535 ns
## 12 x_acf1 150 50 -3.97 109. 1.29e- 4 0.000419 ***
## 13 x_acf10 150 50 0.0701 171. 9.44e- 1 0.944 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ab_decay_impact_het_loading_shift | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.970 | 1.000 | 0.170 | 1.000 | 0.970 | 0.980 | 0.190 | 0.940 | 0.970 | 0.960 |
| 2 | 0.950 | 1.000 | 0.180 | 1.000 | 0.990 | 0.990 | 0.160 | 0.870 | 1.000 | 0.950 |
| 3 | 0.910 | 0.970 | 0.150 | 0.970 | 0.970 | 1.000 | 0.150 | 0.970 | 0.980 | 0.980 |
| 4 | 0.900 | 0.990 | 0.150 | 0.980 | 0.990 | 0.970 | 0.100 | 0.970 | 0.990 | 0.990 |
| rmse | ||||||||||
| 0 | 0.035 | 0.031 | 0.065 | 0.033 | 0.048 | 0.038 | 0.068 | 0.070 | 0.038 | 0.050 |
| 1 | 0.037 | 0.031 | 0.079 | 0.029 | 0.034 | 0.036 | 0.069 | 0.055 | 0.036 | 0.038 |
| 2 | 0.038 | 0.031 | 0.074 | 0.030 | 0.033 | 0.038 | 0.075 | 0.057 | 0.034 | 0.040 |
| 3 | 0.043 | 0.039 | 0.076 | 0.034 | 0.030 | 0.034 | 0.069 | 0.044 | 0.037 | 0.032 |
| 4 | 0.040 | 0.036 | 0.080 | 0.033 | 0.030 | 0.038 | 0.081 | 0.044 | 0.035 | 0.033 |
| bias | ||||||||||
| 0 | −0.014 | 0.005 | 0.026 | 0.011 | 0.036 | 0.011 | 0.018 | 0.064 | 0.009 | 0.037 |
| 1 | −0.022 | −0.009 | 0.034 | −0.003 | 0.016 | 0.012 | 0.018 | 0.047 | −0.003 | 0.021 |
| 2 | −0.021 | −0.006 | 0.035 | −0.001 | 0.009 | 0.001 | 0.039 | 0.049 | −0.004 | 0.019 |
| 3 | −0.029 | −0.026 | 0.027 | −0.019 | −0.003 | −0.008 | 0.015 | 0.034 | −0.016 | 0.004 |
| 4 | −0.025 | −0.022 | 0.034 | −0.016 | −0.001 | −0.006 | 0.031 | 0.034 | −0.013 | 0.006 |
| Notes: | ||||||||||
## [1] "ab_decay_impact_het"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 -0.412 102. 0.681 0.928 ns
## 2 diff1_acf1 150 50 0.0903 78.1 0.928 0.928 ns
## 3 diff1_acf10 150 50 1.23 116. 0.221 0.928 ns
## 4 diff2_acf1 150 50 0.0982 96.8 0.922 0.928 ns
## 5 diff2_acf10 150 50 1.39 116. 0.168 0.928 ns
## 6 e_acf1 150 50 0.377 67.7 0.708 0.928 ns
## 7 e_acf10 150 50 0.112 67.5 0.911 0.928 ns
## 8 entropy 150 50 -0.881 90.5 0.381 0.928 ns
## 9 linearity 150 50 1.40 78.0 0.164 0.928 ns
## 10 spike 150 50 -0.102 98.7 0.919 0.928 ns
## 11 trend 150 50 0.440 95.2 0.661 0.928 ns
## 12 x_acf1 150 50 0.563 90.9 0.575 0.928 ns
## 13 x_acf10 150 50 1.04 94.2 0.302 0.928 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ab_decay_impact_het | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.950 | 0.940 | 0.180 | 0.980 | 0.880 | 0.850 | 0.170 | 0.840 | 0.920 | 0.880 |
| 2 | 0.970 | 0.990 | 0.090 | 1.000 | 0.600 | 0.940 | 0.150 | 0.390 | 0.970 | 0.700 |
| 3 | 0.930 | 0.900 | 0.020 | 0.890 | 0.330 | 0.960 | 0.160 | 0.110 | 0.950 | 0.590 |
| 4 | 0.960 | 0.960 | 0.120 | 0.940 | 0.560 | 0.980 | 0.090 | 0.200 | 0.960 | 0.660 |
| rmse | ||||||||||
| 0 | 0.033 | 0.036 | 0.074 | 0.032 | 0.043 | 0.050 | 0.072 | 0.051 | 0.037 | 0.041 |
| 1 | 0.033 | 0.039 | 0.066 | 0.032 | 0.041 | 0.049 | 0.068 | 0.057 | 0.039 | 0.039 |
| 2 | 0.030 | 0.028 | 0.086 | 0.030 | 0.072 | 0.038 | 0.083 | 0.095 | 0.033 | 0.059 |
| 3 | 0.034 | 0.038 | 0.101 | 0.041 | 0.091 | 0.036 | 0.094 | 0.124 | 0.037 | 0.073 |
| 4 | 0.033 | 0.036 | 0.103 | 0.040 | 0.080 | 0.036 | 0.099 | 0.120 | 0.035 | 0.069 |
| bias | ||||||||||
| 0 | −0.001 | 0.017 | −0.032 | 0.006 | −0.030 | 0.037 | −0.026 | −0.040 | 0.016 | −0.021 |
| 1 | 0.010 | 0.027 | −0.014 | 0.015 | −0.030 | 0.037 | −0.010 | −0.050 | 0.020 | −0.021 |
| 2 | 0.001 | −0.003 | −0.040 | −0.011 | −0.065 | 0.018 | −0.045 | −0.091 | −0.002 | −0.049 |
| 3 | −0.011 | −0.024 | −0.074 | −0.029 | −0.086 | 0.016 | −0.061 | −0.120 | −0.018 | −0.062 |
| 4 | −0.004 | −0.018 | −0.068 | −0.024 | −0.073 | 0.008 | −0.065 | −0.116 | −0.009 | −0.058 |
| Notes: | ||||||||||
## [1] "ab_impact_het_loading_shift"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 -0.911 71.5 0.365 0.515 ns
## 2 diff1_acf1 150 50 -1.93 80.2 0.0569 0.148 ns
## 3 diff1_acf10 150 50 -2.47 75.0 0.0158 0.0514 ns
## 4 diff2_acf1 150 50 -0.854 74.6 0.396 0.515 ns
## 5 diff2_acf10 150 50 -3.03 72.2 0.00339 0.0147 *
## 6 e_acf1 150 50 -3.08 91.3 0.00271 0.0147 *
## 7 e_acf10 150 50 -3.35 93.5 0.00116 0.0147 *
## 8 entropy 150 50 0.272 113. 0.786 0.851 ns
## 9 linearity 150 50 -0.521 89.3 0.604 0.714 ns
## 10 spike 150 50 1.38 90.8 0.171 0.318 ns
## 11 trend 150 50 -0.979 87.1 0.33 0.515 ns
## 12 x_acf1 150 50 -1.63 90.9 0.107 0.232 ns
## 13 x_acf10 150 50 -0.0825 107. 0.934 0.934 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ab_impact_het_loading_shift | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.970 | 0.990 | 0.150 | 0.970 | 0.580 | 0.840 | 0.130 | 0.000 | 0.830 | 0.960 |
| 2 | 0.990 | 0.960 | 0.090 | 0.970 | 0.670 | 0.580 | 0.140 | 0.000 | 0.880 | 0.900 |
| 3 | 1.000 | 0.990 | 0.050 | 0.950 | 0.520 | 0.800 | 0.100 | 0.000 | 0.770 | 0.940 |
| 4 | 0.970 | 0.980 | 0.090 | 0.950 | 0.490 | 0.380 | 0.100 | 0.000 | 0.740 | 0.870 |
| rmse | ||||||||||
| 0 | 0.032 | 0.033 | 0.077 | 0.030 | 0.067 | 0.087 | 0.093 | 0.168 | 0.047 | 0.043 |
| 1 | 0.034 | 0.031 | 0.097 | 0.033 | 0.075 | 0.062 | 0.086 | 0.196 | 0.057 | 0.040 |
| 2 | 0.031 | 0.036 | 0.091 | 0.031 | 0.068 | 0.097 | 0.095 | 0.197 | 0.053 | 0.045 |
| 3 | 0.033 | 0.029 | 0.097 | 0.042 | 0.085 | 0.069 | 0.101 | 0.244 | 0.066 | 0.045 |
| 4 | 0.033 | 0.032 | 0.111 | 0.048 | 0.087 | 0.108 | 0.120 | 0.235 | 0.070 | 0.048 |
| bias | ||||||||||
| 0 | −0.004 | −0.012 | 0.001 | 0.001 | 0.059 | 0.079 | 0.042 | 0.166 | 0.031 | 0.007 |
| 1 | −0.009 | 0.000 | 0.021 | 0.015 | 0.068 | 0.051 | 0.041 | 0.193 | 0.041 | 0.011 |
| 2 | 0.002 | −0.018 | 0.008 | 0.004 | 0.060 | 0.089 | 0.062 | 0.195 | 0.039 | 0.010 |
| 3 | 0.001 | 0.005 | 0.044 | 0.030 | 0.079 | 0.062 | 0.066 | 0.242 | 0.056 | 0.023 |
| 4 | 0.006 | 0.011 | 0.059 | 0.037 | 0.082 | 0.101 | 0.096 | 0.233 | 0.058 | 0.031 |
| Notes: | ||||||||||
## [1] "ab_impact_het"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 0.228 82.7 0.82 0.888 ns
## 2 diff1_acf1 150 50 0.244 79.1 0.808 0.888 ns
## 3 diff1_acf10 150 50 -0.0746 116. 0.941 0.941 ns
## 4 diff2_acf1 150 50 0.264 84.2 0.792 0.888 ns
## 5 diff2_acf10 150 50 -0.311 79.5 0.757 0.888 ns
## 6 e_acf1 150 50 0.283 80.8 0.778 0.888 ns
## 7 e_acf10 150 50 -0.326 78.2 0.745 0.888 ns
## 8 entropy 150 50 -1.78 106. 0.0783 0.494 ns
## 9 linearity 150 50 2.47 80.4 0.0156 0.203 ns
## 10 spike 150 50 -1.16 78.2 0.248 0.654 ns
## 11 trend 150 50 1.04 95.1 0.302 0.654 ns
## 12 x_acf1 150 50 1.11 93.3 0.272 0.654 ns
## 13 x_acf10 150 50 1.59 112. 0.114 0.494 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ab_impact_het | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.990 | 0.970 | 0.100 | 0.980 | 0.990 | 0.980 | 0.180 | 0.810 | 1.000 | 0.980 |
| 2 | 0.960 | 0.990 | 0.160 | 0.980 | 0.840 | 0.900 | 0.150 | 0.550 | 0.950 | 0.900 |
| 3 | 0.950 | 0.990 | 0.090 | 0.960 | 0.890 | 1.000 | 0.250 | 0.520 | 0.980 | 0.930 |
| 4 | 0.970 | 1.000 | 0.140 | 1.000 | 0.990 | 1.000 | 0.080 | 0.950 | 0.980 | 0.990 |
| rmse | ||||||||||
| 0 | 0.032 | 0.033 | 0.072 | 0.032 | 0.034 | 0.038 | 0.072 | 0.078 | 0.032 | 0.033 |
| 1 | 0.028 | 0.034 | 0.068 | 0.031 | 0.034 | 0.046 | 0.073 | 0.085 | 0.032 | 0.031 |
| 2 | 0.035 | 0.031 | 0.060 | 0.033 | 0.053 | 0.064 | 0.065 | 0.110 | 0.036 | 0.043 |
| 3 | 0.038 | 0.042 | 0.080 | 0.044 | 0.050 | 0.037 | 0.066 | 0.114 | 0.036 | 0.046 |
| 4 | 0.032 | 0.031 | 0.075 | 0.031 | 0.032 | 0.034 | 0.067 | 0.076 | 0.035 | 0.034 |
| bias | ||||||||||
| 0 | −0.004 | −0.009 | 0.011 | −0.006 | 0.012 | −0.017 | 0.004 | 0.072 | −0.002 | −0.002 |
| 1 | 0.003 | −0.021 | −0.002 | −0.015 | 0.018 | 0.033 | 0.013 | 0.080 | 0.001 | −0.002 |
| 2 | 0.016 | 0.010 | −0.001 | 0.014 | 0.042 | 0.054 | 0.025 | 0.106 | 0.014 | 0.024 |
| 3 | 0.022 | 0.031 | 0.032 | 0.033 | 0.040 | 0.019 | 0.022 | 0.110 | 0.016 | 0.033 |
| 4 | 0.005 | 0.001 | 0.002 | 0.004 | −0.001 | 0.001 | 0.005 | 0.069 | −0.008 | −0.001 |
| Notes: | ||||||||||
## [1] "ab_no_het_loading_shift"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 2.73 85.0 0.00761 0.00989 **
## 2 diff1_acf1 150 50 -5.94 74.3 0.0000000855 0.00000111 ****
## 3 diff1_acf10 150 50 4.93 96.5 0.00000337 0.0000146 ****
## 4 diff2_acf1 150 50 -3.89 93.4 0.000189 0.000410 ***
## 5 diff2_acf10 150 50 3.15 119. 0.00207 0.00299 **
## 6 e_acf1 150 50 -5.32 75.6 0.00000102 0.00000663 ****
## 7 e_acf10 150 50 -4.85 63.0 0.00000854 0.0000278 ****
## 8 entropy 150 50 0.837 95.5 0.405 0.439 ns
## 9 linearity 150 50 -0.896 112. 0.372 0.439 ns
## 10 spike 150 50 3.75 125. 0.000267 0.000496 ***
## 11 trend 150 50 -3.23 102. 0.00166 0.00270 **
## 12 x_acf1 150 50 -3.89 99.2 0.000179 0.000410 ***
## 13 x_acf10 150 50 -0.544 94.0 0.588 0.588 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ab_no_het_loading_shift | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.980 | 0.980 | 0.160 | 0.980 | 0.930 | 0.920 | 0.190 | 0.900 | 0.960 | 0.920 |
| 2 | 0.960 | 0.940 | 0.120 | 0.940 | 0.970 | 0.960 | 0.120 | 0.950 | 0.940 | 0.890 |
| 3 | 0.950 | 0.950 | 0.100 | 0.950 | 0.970 | 0.970 | 0.170 | 0.970 | 0.970 | 0.840 |
| 4 | 0.940 | 0.940 | 0.070 | 0.930 | 0.990 | 0.980 | 0.190 | 0.980 | 0.980 | 0.900 |
| rmse | ||||||||||
| 0 | 0.033 | 0.029 | 0.076 | 0.030 | 0.033 | 0.037 | 0.058 | 0.082 | 0.032 | 0.037 |
| 1 | 0.033 | 0.031 | 0.085 | 0.031 | 0.036 | 0.046 | 0.066 | 0.089 | 0.036 | 0.035 |
| 2 | 0.033 | 0.035 | 0.082 | 0.036 | 0.031 | 0.035 | 0.067 | 0.073 | 0.037 | 0.042 |
| 3 | 0.036 | 0.037 | 0.083 | 0.039 | 0.030 | 0.035 | 0.061 | 0.071 | 0.035 | 0.046 |
| 4 | 0.036 | 0.036 | 0.086 | 0.037 | 0.028 | 0.035 | 0.059 | 0.077 | 0.032 | 0.041 |
| bias | ||||||||||
| 0 | −0.010 | 0.002 | 0.003 | −0.005 | 0.011 | 0.010 | −0.004 | 0.074 | 0.004 | −0.016 |
| 1 | −0.007 | 0.001 | 0.021 | −0.001 | 0.018 | 0.027 | 0.014 | 0.083 | 0.004 | −0.007 |
| 2 | −0.005 | −0.015 | 0.017 | −0.015 | 0.001 | −0.001 | 0.011 | 0.063 | −0.005 | −0.026 |
| 3 | −0.021 | −0.024 | −0.000 | −0.026 | −0.007 | −0.010 | −0.002 | 0.061 | −0.012 | −0.035 |
| 4 | −0.016 | −0.023 | −0.007 | −0.024 | −0.001 | −0.002 | 0.001 | 0.068 | −0.009 | −0.028 |
| Notes: | ||||||||||
## [1] "ab_no_het"
## # A tibble: 13 x 8
## vars n1 n2 statistic df p p.adj p.adj.signif
## <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 curvature 150 50 0.280 79.6 0.78 0.922 ns
## 2 diff1_acf1 150 50 0.0881 88.6 0.93 0.986 ns
## 3 diff1_acf10 150 50 -0.0174 73.5 0.986 0.986 ns
## 4 diff2_acf1 150 50 0.775 84.5 0.44 0.914 ns
## 5 diff2_acf10 150 50 -0.497 69.7 0.621 0.914 ns
## 6 e_acf1 150 50 -0.559 83.3 0.578 0.914 ns
## 7 e_acf10 150 50 -1.46 83.8 0.148 0.914 ns
## 8 entropy 150 50 -0.883 95.6 0.379 0.914 ns
## 9 linearity 150 50 2.65 86.4 0.00947 0.123 ns
## 10 spike 150 50 -0.351 81.6 0.726 0.922 ns
## 11 trend 150 50 0.768 90.4 0.445 0.914 ns
## 12 x_acf1 150 50 0.479 93.3 0.633 0.914 ns
## 13 x_acf10 150 50 1.11 98.6 0.271 0.914 ns
| Metrics by Method | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ab_no_het | ||||||||||
| Method | gsynth | sdid | sdid_debiased | sdid_uncon | scm | gfoo | gfoo_debiased | mc | causalimpact | ensemble |
| coverage | ||||||||||
| 1 | 0.990 | 0.950 | 0.140 | 0.950 | 0.980 | 0.820 | 0.160 | 0.990 | 0.950 | 0.760 |
| 2 | 0.980 | 0.950 | 0.170 | 0.920 | 0.980 | 0.770 | 0.180 | 0.940 | 0.960 | 0.620 |
| 3 | 0.950 | 0.890 | 0.100 | 0.880 | 0.980 | 0.830 | 0.190 | 0.950 | 0.910 | 0.750 |
| 4 | 0.950 | 0.900 | 0.210 | 0.850 | 0.980 | 0.900 | 0.170 | 0.890 | 0.930 | 0.740 |
| rmse | ||||||||||
| 0 | 0.033 | 0.041 | 0.072 | 0.040 | 0.036 | 0.043 | 0.065 | 0.084 | 0.039 | 0.066 |
| 1 | 0.030 | 0.042 | 0.064 | 0.039 | 0.029 | 0.053 | 0.061 | 0.103 | 0.034 | 0.060 |
| 2 | 0.033 | 0.044 | 0.070 | 0.043 | 0.033 | 0.060 | 0.064 | 0.111 | 0.039 | 0.067 |
| 3 | 0.033 | 0.048 | 0.065 | 0.047 | 0.029 | 0.053 | 0.064 | 0.126 | 0.038 | 0.058 |
| 4 | 0.035 | 0.051 | 0.074 | 0.049 | 0.032 | 0.056 | 0.074 | 0.132 | 0.037 | 0.053 |
| bias | ||||||||||
| 0 | −0.002 | −0.026 | 0.017 | −0.024 | −0.010 | −0.027 | −0.008 | 0.078 | −0.015 | −0.056 |
| 1 | 0.002 | −0.032 | 0.005 | −0.028 | −0.007 | −0.042 | −0.016 | 0.099 | −0.014 | −0.053 |
| 2 | −0.008 | −0.033 | 0.003 | −0.031 | −0.013 | −0.050 | −0.022 | 0.108 | −0.022 | −0.059 |
| 3 | −0.011 | −0.039 | −0.006 | −0.037 | −0.003 | −0.041 | −0.016 | 0.122 | −0.022 | −0.049 |
| 4 | −0.013 | −0.040 | −0.014 | −0.038 | 0.004 | −0.044 | −0.016 | 0.128 | −0.015 | −0.042 |
| Notes: | ||||||||||